The two lines above, L335:336, is one of them meant to be an aggregate metric of all interfaces on the host? If so it would be ideal if we could, at least by default, collect only the total usage or the per interface metrics.

That's what the [host|vm|cluster|datastore]_instances flag does. If you set it to false, you'll get the behavior you're asking for. By default, they're set to "true", except for datastores. This reflects some kind of "best practice" of what people usually want to collect.

Do you think we should add support for collecting only the instance metrics? If you collect instance metrics then there is less reason to store the totals since it can be computed at query time. Perhaps the instance variables could be something like: `vm_metric_mode = "instance|aggregate|all".

One thing that is a moderate issue with metrics tagged like this is selecting only the instance without the totals. For example say we have these metrics:

vsphere_host_net,interface=vmnic0 bytesTx_average=1i
vsphere_host_net,interface=vmnic1 bytesTx_average=3i
vsphere_host_net bytesTx_average=2i

To average only the instance versions you need to know the secret method to remove series containing the total aggregation:

select mean(bytesTx_average) from vsphere_host_net where interface =~ /./

What we usually do is try to rename the field by appending the aggregation type, the new field name ensures the values do not get aggregated together:

vsphere_host_net,interface=vmnic0 bytesTx_average=1i
vsphere_host_net,interface=vmnic1 bytesTx_average=3i
vsphere_host_net bytesTx_average_average=2i

So that would be very ugly, maybe something like this would be better?:

vsphere_host_net,interface=vmnic0 bytesTx_average=1i
vsphere_host_net,interface=vmnic1 bytesTx_average=3i
vsphere_host_net total_bytesTx_average=2i

I see your point that it's hard to search to things that are missing a certain point tag. The problem is that it's tricky to reliably determine whether a metric in vCenter will ever show up with an instance metric, so you'd have to add the "total" or "average" to anything that's not a metric on an instance. You never know if that same metric is going to show up in the future with an instance. @puckpuck what do you think?

E.g. cpu.blablabla doesn't have an instance metric in vSphere version N, so all metrics are reported as just cpu.blablabla. In version N+1, the metric is given an instance and all of a sudden cpu.blablabla becomes cpu.blablabla.total. I really don't like that.

Yeah, that would be a downside, you don't want to switch your dashboards because you change this minor setting. It seems to me that the best solution would be to collect only one type or the other, at least by default. Don't have to worry about excluding data from the query if you don't have it, and I think having both forms is redundant anyway.

I think the best solution is to always emit the instance tag and send it as instance-total for the aggregate metrics (i.e. metrics without an instance tag).

That would be similar to the way e.g. input.cpu does it.

That will work.

Ok. Commit coming shortly.

Can you describe the relationship between cluster, esxhostname, source, moid, and vcenter

A vCenter has clusters, clusters have esxhosts (the physical machines) identified by the esxhostname, hosts have vms. The source field is the name of whatever object is collected (vm, host, cluster or datastore). A moid is the unique internal id of any object in a vCenter. It is sometimes useful for uniquely identifying an object when e.g. it has been renamed.

Do you think we need this in the README? My assumption was that most people using thisa plugin would already be familiar with this.

(MOID stands for Managed Object ID and is a well-known property of vSphere resources)

Do you think we need this in the README?

No probably not, I agree that anyone monitoring will be more up to speed than I am.

I think this is a couple collections, lets remove it down to just one interval.

OK. Will check on that.

Some of the metrics in the default config are not documented in METRICS.md

I don't see collect_concurrency being used, do we still need it?

The resource that is scarce is connections, so I propose we rename this max_discover_connections.

BTW, did you see they added an option for this to the transport in Go 1.11 (MaxConnsPerHost)? Should be pretty helpful, but we still target Go 1.9 so we can't use it yet.

I will probably merge the Tags section with Measurements & Fields when I merge the PR, unless you do it first :)

Check EXAMPLE_README.md for the latest style.